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Abstract 

University administrators rely on institutional research to provide information to guide their decisions. In times of 
rapid change, decision-makers often need new kinds of information, and they need it quickly. Institutional research 
has long relied on survey research to learn about student experiences. This paper describes and evaluates two 
methods of using electronic mail to gather information from students about their experiences. The conclusions draw 
heavily from case studies in electronic surveying conducted during Spring 1996 and Fall 1997 at a 5,000-student 
public comprehensive university. The study provides concrete examples of putting new communications 
technologies to work for institutional research and exposes their strengths and weaknesses relative to traditional pen- 
and-paper mail surveys. 
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Between Anecdote and Science: Using E-maii to Learn about Student Experiences 

University administrators rely on institutional research to provide information to guide their decisions, and 
institutional research has long relied on survey research to learn about student experiences. Universities administer 
surveys to learn why students come, why they stay, and what they take with them when they go. Survey 
instruments range in sophistication from a few open-ended questions to long questionnaires answered on computer- 
scannable forms. In most situations, a well-designed survey will take several months to develop, administer, code 
and analyze. Surveys conceived of this spring will most usually not affect programs or policies until a year or more 
in the future. 

In times of rapid change, decision-makers often need new kinds of information, and they need answers 
quickly. Surveys need content flexibility and quick turn-around time from problem identification to program re- 
design. In this information age, institutional research can gain flexibility and time-savings by moving some survey 
functions onto the computer. This paper describes and evaluates two methods of using electronic mail to gather 
information from students about their experiences. The conclusions draw heavily from case studies in electronic 
surveying conducted during Spring 1996 and Fall 1997 at a 5,000-student, public, comprehensive university. The 
study provides concrete examples of putting new communications technologies to work for institutional research, 
and exposes their strengths and weaknesses. 

The first section explains some of the potential benefits of using e-mail to conduct survey research, with 
references to relevant literature on the topic. The second section describes how to use an e-mail distribution list to 
distribute questions and collect responses. The third section describes how to use a hyper-text mark-up language 
(HTML) form posted to an internet web site to conduct survey research. The conclusion evaluates the strengths and 
weaknesses of these two methods relative to each other and relative to traditional pen-and-paper surveys. 

Why Consider Moving Surveys “On Line”? 

Many, especially larger, offices of institutional research have developed well-oiled survey machines. We 
know how to move a new instrument quickly through the review board, we know how long it’s going to take the 
printer to make the copies, and we know our administrative assistants can fold and stuff envelopes in their sleep. 
Why bother to think about changing? Technologies, respondents and demands change, and so should we. 

Technological change has opened up a new way for researchers to reach respondents. Of course, each 
technology has strengths and weaknesses. Phone surveys are better for some topics and survey designs (for example 
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if there is a lot of branching), but a socially sensitive topic might generate more truthful responses through a fully 
anonymous mail survey. When is the time right to move a design on-line? In a literature review of electronic mail 
survey research, Liz Thach (1995) identifies at least four characteristics of the method that seem especially appealing: 
(T)he speed of the message transmission could result in faster response rates; asynchronous 
communications allow the users time to think about their answers; lack of intermediaries will increase the 
chance of the respondents receiving the survey immediately...; and the ephemeral quality of messages may 
encourage the user to respond in a more candid fashion... (p. 27-28). 

Some research has shown that respondents react differently to electronic mail surveys than traditional 
methods. Sproull (1986) found that electronic mail surveys had response rates 20% higher than paper surveys, but 
lower than interviews or phone surveys. Kiesler and Sproull (1986) found that electronic mail survey respondents 
completed more items and made fewer mistakes than respondents completing a similar paper instrument. Several 
researchers have found that respondents answer with more socially undesirable responses when using a computer 
than when completing paper questionnaires or participating in interviews (Kiesler and Sproull, 1986; Sproull, 1986; 
Synodinos and Brennan, 1988; Walsh et. al., 1992). However, others have found that “computers yield similar 
responses to those obtained using traditional paper instruments” (Rosenfeld, et. al., 1993). 

A sample’s reaction to computerized instruments most certainly relates to characteristics of that sample. 

For example, research has found that respondents with high computer anxiety have lower response rates to 
computerized surveys than to traditional methods. A survey of students who wait until their senior year to fulfill a 
computer skill requirement might best be distributed through the mail! On the other side of the coin, young people 
who have grown up with computers often type much faster than they write (and, certainly, more legibly!). They 
may provide more complete answers to open-ended items, or will at least perform the survey tasks more quickly. 

Higher education may be uniquely positioned to develop electronic mail surveying because many of our 
student populations do not face the kinds of access barriers to computers that other organizations must confront. At 
most institutions, students, faculty and staff get free access to electronic mail; computer centers often offer free or 
inexpensive training sessions to novices; many schools offer computer-based writing courses; faculty are increasingly 
asking students to communication with them and with each other through electronic mail. Indeed, some may argue 
that students who graduate today and have not learned to rely on electronic mail may not have been properly 
prepared for the post-college world they now confront! 
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Changes in the nature of demands placed on institutional research provide a final set of reasons why 
electronic mail surveying may make sense. Many aspects of our organizations increasingly turn to modem “total 
quality” management techniques as guides to improved performance. These modem techniques require collecting 
information and making decisions on the basis of it, and relying less on intuition, subjective assessments and 
anecdotes. Institutions are asking new kinds of questions about their services and achievements, and want to make 
decisions based on evidence, not hunch. An important dimension to the demands placed on institutional research 
relates to cost. Decision-makers want institutional research to collect new policy-guiding information within the 
context of current (and often shrinking) budgets. 

Surveying through electronic mail can save time and money when compared to traditional methods. An 
electronic mail survey requires no paper, no envelopes, no printing, and no postage. Time spent printing or 
copying the instrument, folding and stuffing envelopes, and sorting bulk mailings into proper order disappears. 

Mail transit time for surveys to reach their destinations and return back to the researcher’s hands falls from days and 
even weeks to seconds. Steinfield (1990) gives an example of time-savings from a survey done at a corporation: 
They broadcast a message throughout the company asking for advice.... The message included a 
description of the options, and what they saw as the strengths and weaknesses of each... This message was 
sent in the morning, and by late afternoon they had received several hundred replies. The group then 
tallied up the replies, and sent a message back to everyone in the same afternoon.... Without electronic 
mail, this same activity simply could not have occurred without great cost and effort, and yet in this 
instance was completed all in the same day (p. 288). 

Another difference relates to the researcher’s control over the eventual sample size. Most designs for mail 
and phone surveys specify a limited number of attempted contacts with potential respondents. For example, a 
potential mail survey respondent might get a first wave, a postcard reminder and a second wave; a phone survey 
protocol might specify making five attempts to reach a respondent. Researchers often find it difficult to alter the 
design or protocol in the middle of the administration based on initial results. In contrast, an electronic mail 
process gives the researcher more opportunities to alter the process based on initial responses in order to achieve an 
optimal sample size. For example, if an organization wants to base its analysis on 500 responses, the researcher can 
randomly select 1,000 individuals to participate. If that group fails to provide the required number of responses, 
additional individuals can be surveyed. Of course, this will not work well for time-sensitive issues, and the 
researcher must compare different samples for homogeneity. On the other hand, if 500 or more of the original group 
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responds, the research can drop plans for a second wave. (A planned second wave of a mail survey would almost 
always be carried out because duplicating, paper and envelope costs have already been sunk into the project from the 
outset.) The researcher can even alter the instrument between waves, if needed, to correct errors not found through 
pilot testing. 

Finally, electronic mail surveys can save time and improve accuracy at the data entry stage. Essentially, 
the respondents enter the data directly into the form needed for analysis. The researcher simply moves the responses 
from in electronic mail folder into the data base. (In some systems, a computer programmer can help eliminate even 
that step.) Time and accuracy issues are especially prevalent with open-ended questions. In traditional methods, 
somebody must decipher and type out the responses to open-ended questions before the researcher can begin 
analyzing their content. With electronic mail surveys, virtually all of the time formerly spent doing date entry 
becomes time available for analysis. 

Of course, electronic surveys can not and should not fully replace surveys all the time. After discussing 
two different methods of conducting electronic mail surveys, this paper will conclude with some practical advice 
about when these newer methods should be used and when they should be avoided. 

Electronic Mail Surveying with Distribution Lists 

One way to conduct an electronic mail survey requires setting up a distribution list containing all members 
of the target population or sample. The researcher then sends the survey questions as an electronic mail message to 
all potential respondents. The message instructs them to select the “reply” option, type their response, and select 
the “send” option. When preparing to conduct an electronic mail survey this way, the researcher should go through 
the following steps. First, carefully word the instrument. The message respondents get must start with a simple 
description of the survey’s purpose, information about confidentiality, and a time limit on responding. The 
message must contain extremely clear, installation-specific instructions. For example, if the electronic mail software 
used by most students uses the word “Transmit” as the command for sending the message, the instructions should 
use that language as well. 

A distribution list technique is especially well suited for asking a few, carefully-worded, open-ended 
questions. This technique does not work well for items that have a fixed response set because the researcher has no 
control over where the respondent will place the answer. That is, answers can come in left justified, right justified, 
centered, single spaced or double spaced; some respondents will place all their answers after the questions, while 
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others will intersperse their answers with the questions. This variability detracts little from the ease of analyzing 
open-ended items, but can make analyzing closed items quite tedious. 

A second thing to do when preparing a distribution list survey is to decide what other information you 
need to know about respondents besides their answers to the questions. Ideally, the researcher should prepare a file 
that matches students’ electronic mail addresses with the tracking number that identifies students in the institution’s 
data base. This will enable the researcher to merge responses with relevant demographic and academic records, and 
eliminates the need to include questions like gender, ethnicity, class standing or major, for example. 

Third, the researcher needs to prepare the distribution list and the distribution protocol. In general, this 
will require working closely with personnel in the computer center to identify the most efficient way to send a large 
group of people the same message simultaneously. Of course, the computer center is also the logical first place to 
contact to learn students’ electronic mail addresses. The process of constructing the distribution list will vary 
depending on the hardware and software in use at each institution. Many researchers will learn that the computer 
center actively maintains an “all student” distribution list. 

In order for students to respond by selecting the reply option, the initial message needs to be mailed from 
the electronic mail account where the researcher wants to collect the responses. The researcher should consider 
establishing a special account for the survey in order to keep survey responses separate from other in-coming 
electronic mail messages. The protocol should also include plans about conducting follow-ups. Ideally, the 
researcher should learn how to remove people from the distribution list once they have responded so they will not 
receive a second wave. The researcher needs to decide how long to wait before mailing out the second wave and any 
subsequent waves. The computer center may be able to provide a frequency distribution of how many days have 
passed since registered users last signed on, which indicates approximately how often people might be reading their 
electronic mail. For example, if the computer center indicates that 80% of the people signed on yesterday and 10% 
last signed on two days ago, the researcher could feel comfortable mailing follow-up waves approximately a week 
apart. On the other hand, if the user statistics indicate that most users only sign on approximately once a week, the 
researcher may want to wait two weeks before sending the follow-up wave. 

Once responses start coming in, the researcher needs to know what to do with them. It might be ideal to 
set aside some time each day during the survey process to move the in-coming responses out of the electronic mail 
folder and into a more permanent file. For some very simple questions (i.e. “Should our mascot be the Indians or 
the Icebreakers?”), the researcher can read and code the response and then delete it. When a response is processed, 
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that respondent’s address should be removed from the distribution list. It may be useful to format each 
respondent’s response into a single paragraph, with the identification number as the first element of the paragraph so 
that the responses can be sorted and more easily merged with other relevant data. As with any open-ended survey 
question, the researcher needs to prepare to do a lot of content analysis. 

A Case Study of the Distribution List Method: The “Feedback Project” 

During Spring 1996, the Office of Institutional Research at the University of Wisconsin - Green Bay 
conducted a ten-week paperless survey - the “Feedback Project” - to collect information from students about a 
variety of educational issues. A distribution list of approximately 2,400 full-time undergraduates was developed. 
The first week, all members on the list received a message that described the purpose of the project and asked them 
to reply with the word “Yes” ifthey wanted to participate, and to simply delete the message if they did not want to 
participate. A new distribution list of approximately 450 willing participants was prepared. Then, each Monday for 
the next eleven weeks those students received a single open-ended question, along with instructions when and how 
to reply and how to be removed from the project. The subject field of the electronic mail message was used to 
indicated the question number, which facilitated keeping responses separate since students were typically given two 
weeks to respond to each item. Approximately two weeks after posting each item, all responses were gathered into a 
single word processing file. Responses were coded into categories relevant to that question. Short, one or two-page 
summaries were written and posted to the institutional research office’s web site and distributed to campus leaders. 
At the end of the project, participants received a “thank you” message that indicated how they could access the 
summaries. 

Table 1 contains more detailed information about the questions asked in the Feedback Project. Topics 
touched upon included pedagogy (weeks 1, 5 and 6), institutional mission (weeks 2 and 8), enrollment 
management (weeks 3, 7, 9 and 1 1), student life (week 4), and outcomes (week 10). Some questions were timed 
to coincide with specific events. For example, students received the question about time management during the 
midterm exam period, and they received the question about course availability the week advanced registration began. 
Each week, approximately one-third of the project participants responded to the question. The average length of the 
responses ranged from 69 to 103 words. Because the Feedback Project was experimental and exploratory, responses 
were stored anonymously and no second waves were conducted. The content of each electronic mail response was 
copied to a word processing file, and the address information was purged. This prevented merging the responses 
with information from the student data base. 
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Table 1. Feedback Project Summary 



Week 


Topic 


Question 


Respondents 


Mean 

Words 


1 


Pedagogy 


What are the best courses you’ve ever taken at UW- 
Green Bay? What, specifically, was so outstanding 
about them? 


182 | 


96 


2 


Mission 


What does “interdisciplinary” mean to you? How 
has it affected your studies at UW-Green Bay? 


158 


84 


3 


Enrollment 

Management 


Do you know any students who attended college in 
the fall and aren’t in school now? What are they 
doing? Do you think they made the right choice? 


105 


95 


4 


Student Life 


How many hours a week do you work or spend 
caring for family members? How do these work and 
family commitments affect your education? 


159 


80 


5 


Pedagogy 


What do you think about UW-Green Bay’s General 
Education requirements? Has taking General 
Education courses been a positive experience for you? 
Explain. 


159 


90 


6 


Pedagogy 


What courses and/or assignments at UW-Green Bay 
have had the most positive impact on your ability to 
write? 


120 


70 


7 


Enrollment 

Management 


Many students used to graduate in four years, but 
now most take longer. How long will it take you, 
and why? 


184 


81 


8 


Mission 


What do you think about the Senior Seminar 
requirement? Is it a good requirement, or not? 
Explain. 


139 


69 


9 


Enrollment 

Management 


Have you ever gone to register for a course and found 
out it wasn’t offered or was already closed? How did 
you handle the situation? How has course 
availability at UW-Green Bay affected you? 


139 


103 


10 


Outcomes 


What do you plan to do after graduating from UW- 
Green Bay? [Seniors only! 


29 


Not 

available 


11 


Enrollment 

Management 


Do you plan to attend UW-Green Bay next fall? 
Why, or why not? fNon-seniors only! 


126 


Not 

available 



Electronic Mail Surveying with HTML Forms 

In the past few years, thousands of colleges and universities have begun posting information about their 
institutions to the World Wide Web. Although organizations most commonly use the World Wide Web to 
provide information to various constituents, it is also possible to use the Web to collect information. Web forms 
let a reader send some information back to a Web server for some particular action. In the following discussion, the 
method described collects information from a reader and asks the server to send the formatted results to the 
researcher’s electronic mail account. Additional programming could ask the server to send the results directly into a 



data file. 
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Web forms consist of special commands embedded into a hypertext markup language (HTML) file (NCSA, 
1996). The commands in table 2 initiate an HTML document, specify a fill-out form, direct the form to send the 
results to an electronic mail account, structure a single question, and direct the respondent to package up the results 
and post them to the server. The actual HTML file would look something like figure 1, and would appear on a 
browser screen as shown in figure 2. 



Table 2. Simple HTML Form Commands 



Commands 


Comments 


<html> 


This initiates the HTML document. 


<head><title>Simple Form</title></head> 


The title “Simple Form” will appear on the user’s browser 
screen. 


<body> 


This initiates body of the HTML document. 


<form 

method=“post” 

action- 7htbin/cgi-mailto/ 

vour-emailfSfyour.university.edu’^ 


This initiates the form and indicates results will be posted to 
the account specified in the “action” details. 


Indicate your sender: 


These directions appear on the screen. 


<input type=“radio” 
name=“Gender” 
value=“M” checked> 
Male 


There are several input types. This variable is named 
Gender, and can have two values (M or F). M is 
automatically selected. 


<input type-“radio” 

name=“Gender” 

value=“F”> 

Female 


The words “Male” and “Female” appear on the screen to the 
right of the selection buttons. 


<p> 


This causes a blank line to appear between the gender item 
and the submit directions. 


When done, click on submit: 


These directions appear on the screen. 


<input type=“submit” 
value-“Submit”> 


A button with the word “Submit” appears on the screen. 
When selected, the results are posted. 


</form> 


This ends the form. 


</body> 


This ends the body of the HTML document. 


</html> 


This ends the HTML document. 



The researcher can select from three main kinds of input types to structure responses: radio, checkbox and 
text. (A fourth kind of input, called select, is not described in this paper.) The Simple Form illustrated in figures 
1 and 2 uses a “radio” input. Radio-type input forces users to make one selection from a set of choices provided 
with the same name on the input tag. The checkbox-type input is a single choice. It is either yes, or no; on or off. 
It is appropriate for a list where respondents are asked to “Check all that apply”. The text-type input permits the 
respondent to provide comments to an open-ended question. On the input tag, after indicating the type of input 
sought, additional subcommands structure the response, as shown in table 3. 



O 
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Figure L Simple HTML Form File 

<html> 

<body> 

<head><title>Simple Form</ titlex/head> 

<forru method="post" 

action=" /htbin/cgi-mailto/ furlongd@gbms01 . uwgb . edu"> 
Indicate your gender: 

< input type= "radio" name = "Gender" value="M" checked>Male 

< input type="radio" name = "Gender" value= "F">Female 

<P> 

When done, click on the submit button: 

< input type= " submit " value= " Submit " > 

</form> 

</body> 

</html> 



Figure 2. Appearance of Simple HTML Form on Browser 









Netscape: Simple Form 
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When done, click on the submit button: [ Submit ] 
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(Exact appearance varies from system to system.) 
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Table 3. Input Types and Their Sub-commands 



INPUT= 


NAME= 


VALUE= 


Other subcommands 


“radio” 


“variable name” 


“Value” The value of the 
selected choice will appear in 
the data file. 


CHECKED If this appears after 
a value, that value will be treated 
as the default. 


“checkbox” 


“variable name” 


“Value” The value will appear 
if the item is selected, otherwise 
the variable will be blank. 


CHECKED If this appears after 
the value, the default for the 
checkbox is selected (as opposed 
to empty). 


“text” 


“variable name” 


“Value” The value will appear 
as the default. Value is 
optional for text-type input. 


MAXLENGTH=xx This 
determines the maximum 
number of characters accepted as 
input. 



While Input=“text” works well for open-ended items that anticipate short, or single-word responses, the 
textarea tag works better for longer answers. Textarea tags have three subcommands: name, the variable’s name; 
rows, the height of the text entry field; and cols, the width in characters of the text entry field. The dimensions of 
the field on the screen do not have any bearing on the amount of text respondents can provide: textarea tags 
automatically have scroll bars and accept unlimited amounts of text. Unlike the input tag, the textarea tag requires a 
terminating, or /textarea tag. The tag below would create an entry field four rows tall and 25 columns wide for 
respondents to write about their plans: 

<textarea name=“PIans” rows=4 cois=25></textarea> 

After the user completes the form and presses the submit button, the file server packages the response and 
sends it to the researcher’s electronic mail address. Figure 3 shows what the response to the Simple Form might 
look like in the researcher’s in-coming electronic mail account. 



Figure 3. Structure of Response to Simple Form Example 



From: IN% " HTTP_SERVER@university . edu" 21-APR-1997 12:20 
To : IN%"your . address@university.edu" 

CC: 

Subj : 

(Other electronic mail processing details, varying 
according to the system used) 

REMOTE_ADDRESS : 143.200.30.67 
Gender : F 



O 

ERIC 
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As with surveying with a distribution list, when using an HTML form the researcher must do much more 
than simply prepare the instrument. Two important decisions relate to controlling the sample and processing the 
results. An HTML survey could potentially sit on a university’s Web site for weeks without having a respondent 
find it. Or, an instrument designed for students could gather responses from faculty and staff. Figuring out how to 
control the sample is critical. If the instrument is positioned in a hart-to-miss location — such as being linked with 
a flashing arrow to the institution’s Home Page — the researcher must decide if it is important to track who 
responds. Necessary demographic questions can be added into the form, or a student identification can be requested. 
If identification is requested, respondents should be informed that their answers are not anonymous. An alternate 
method of distribution involves using some other means to inform potential respondents how to find an otherwise 
difficult to locate survey. For example, instructors can announce the http: address of the instrument in relevant 
classes. 

A second set of issues surrounds handling responses. Because the forms process enables the researcher to 
collect answers to closed-ended questions in a very structured way, it is quite easy to use statistical packages like 
SAS and SPSS to tally up the answers. For example, if a survey contains five items, each electronic mail response 
will contain five lines. The responses to the first item will always appear on the first line. The first element on 
each line will be the name of the relevant variable, which will be followed by a colon, a space, and the value the 
respondent selected for that item. The researcher would collect the electronic mail responses, concatenate the data 
into one long file, and write a SAS or SPSS program to read the data. Each case would consist of five records, or 
lines, and each variable’s location could be easily identified. Including multiline textarea responses along with the 
closed-ended questions somewhat complicates the process. Because users have the option of typing a lot or a little, 
the in-coming electronic mail responses no longer have identical structures. (However, it is possible to sort out the 
open-ended text and remove it from the file before processing the rest of the information with a statistical package. 
And, it is possible to use word processing macros to restructure the text in such a way that a statistical package can 
be used.) 

A Case Study of the HTML Form Method: The “First-year Survey” 

In Fall 1996, the Counseling Center and Office of Institutional Research at the University of Wisconsin - 
Green Bay conducted a survey of first-time freshmen using an HTML form posted to the institution’s Web site. 

The instrument sought to learn how new students handled the transition from high school, or work and family, to 
college, and to evaluate some of the things the institution does to facilitate that transition. The form contained 
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twelve closed-ended questions and ten open-ended questions. Initial drafts of the survey contained sensitive 
questions about alcohol use and sex practices. However, the research design did not work well for such private 
questions. First of all, students completed the form during class time. More critically, students responses remain 
visible on the computer screen until all portions are complete and the form is submitted. It would be very easy for 
students to see each others’ answers, even unintentionally. The final questionnaire tried to keep the public nature of 
the distribution method in mind. 

Figures 4 and 5 contain a portion of the instrument. The complete survey can be viewed at 
http://www.uwgb.edu/~fiirlongd/fy.html. As with the Feedback Project, this survey was experimental and 
exploratory and no identification was collected or tracked. In November 1996, researchers asked instructors teaching 
freshmen writing to have their students complete the survey as part of a class exercise on using the Internet. The 
form was made available to the writing classes for a two-week period in December. Students from seven writing 
sections submitted 106 usable responses. 

The responses were concatenated into a single word processing file which was processed two ways. First, 
the file was read with a statistical package to generate simple frequency distributions for the closed items. Second, 
a macro was written to remove unnecessary paragraph breaks so that each variable could be treated like a single 
paragraph. This permitted sorting the file by paragraphs, and then grouping all of the open-ended responses to a 
particular item together to facilitate an analysis of their content. For the closed items, 104 of the 106 surveys were 
complete, which might indicate that respondents did not have any problems completing the task. The open-ended 
responses contained no systematic error patterns that would indicate students experienced technical difficulties 
working with the textarea boxes. Two instructors who administered the survey reported that some students could 
not submit their instrument, but no technical explanations for the problem ever emerged. By the end of December, 
researchers had completed the frequency tallies and content analysis, and had written a report to senior 
administrators. The complete tum-around time for this project was less than one semester. 



Strengths and Weaknesses: HTML vs. Lists vs. Traditional Methods 
This concluding section will summarize the relative strengths of HTML form and distribution list 
electronic mail surveys, and will then address some weaknesses of those techniques relative to more traditional 
methods. Some of the factors one might consider when choosing between HTML form and distribution list surveys 
pertain to technical skills, the topic and predominant question type, and sample bias. 
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Figure 4. HTML Code for Part of First-Year Survey 



<html> 

<headxtitle>First-Year Student Feedback Request</ title></head> 
<h2>First-Year Student Feedback Request</h2> 

<pxhr> 

Please help us better serve new students in the future by telling us about 
your experience as a new student at the University of Wisconsin-Green Bay. 
All responses will be compiled in such a way as to keep your answers 
anonymous. Thank you for your help.<hr> 

<a name=" Submit " x/a> 

<form method="post " action=" /htbin/cgi-mailto/counsel@gbms01 . uwgb.edu"> 
<p>What is your student status in Fall 1996? 

<input type=" radio" name= M Vl" value= u NF" checked>New Freshman 
< inpu t type= " r adi o " name= " VI " va lue= " TR " >Trans f er 
<pxhrxcenterxh3>I. ACADEMIC EXPERIENCE</H3x/CENTER> 

<pxb>So far, what do you like MOST about your classes at UW-Green Bay?</b> 
(When entering text, you <i>may</i> press the return key to start new lines 
if you would like. You may type as much as you like, using as many lines as 
you would like.) 

<blockquote><textarea name="V2" rows=4 cols=60x/ textareax/blockquote> 
<pxb>What do you like LEAST about your classes at UW-Green Bay?</b> 
cblockquotextextarea name="V3" rows = 4 cols=60></ textareax/blockquote> 

_______ (portion skipped here) ------- 



<b>How important do you think it is for UW-Green Bay to provide 
first-year students with the following information?</bxp> 

1 = Very Important<br> 

2 = Important<br> 

3 = Neither important nor unimportant<br> 

4 = Unimpor tant<br> 

5 = Very Unimportant<br> 

<pxol> 

<lixb>Campus rules and regulations : </b> 

<input type="radio" name="vlO" value="l">l 

1 name="vlO" value="2" >2 
1 name="vlO" value="3">3 
1 name="vlO" value=" 4" >4 
1 name="vlO" value= " 5 ">5<brxp> 



<input type= "radio " 
<input type= " radio 11 
<input type=" radio" 
<input type=" radio" 



<lixb>Alcohol/drug potential for abuse :</b> 



<input type= " radio " 
<input type=" radio" 
<input type=" radio" 
<input type= "radio " 
<input type=" radio" 



name= " vll 1 
name= " vll ' 
name= " vll " 
name= " vll ' 
name= " vll ' 



value= " 1 " >1 
value= " 2 " >2 
value= " 3 ">3 
value= " 4 " >4 
value= " 5 " >5<brxp> 



------- (portion skipped here) ------- 

<hrx center xh3>THANK YOU FOR YOUR FEEDBACK! </H3x/CENTER> 
When your survey is complete, click on the submit button: 

< input type= " submi t " value= " Submit * > 

</form> 

</body> 

</html> 
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Figure 5. Appearance of Part of First-year Survey on Browser 






IE 



Netscape: First-Year Student Feedback Request 









Back 



Forward 



Home 



Reload 



Images 



Open 



Print 



Find 



Stop 



N 



First-year Student Feedback Request 



Please help us better serve new students in the future by telling us about your experience as a new 
student at the University of Wisconsin-Green Bay. All responses will be compiled in such a way as 
to keep your answers anonymous. Thank you for your help. 

I. ACADEMIC EXPERIENCES 

So far, what do you like MOST about your classes at UW-Green Bay? (When entering text, you 
may press the return key to start new lines if you would like. You may type as much as you like, using 
as many lines as you would like.) 



What do you like LEAST about your classes at UW-Green Bay? 



• (portion skipped here) • 



How important do you think it is for UW-Green Bay to provide first-year students with 
the following information? 

1 = Very Important 

2 = Important 

3 = Neither Important nor Unimportant 

4 = Unimportant 

5 = Very Unimportant 



$ 



1. Campus rules and regulations: 1 

2. Alcohol/drug potential for abuse: 

(portion skipped here) 



3 

2 



THANK YOU FOR YOUR FEEDBACK! 
When your survey is complete, click on the submit button: 



Submit 



BEST COPY AVAILABLE 
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The technical skills of both the researcher and the university community have a bearing on method choice. 
Using an HTML form requires that the institution be capable of mounting web pages and be supportive teaching 
people how to use the internet. From the researcher’s perspective, this method requires learning about HTML and 
web page construction, or developing some staunch allies in the computer center. Doing distribution list surveys 
also has technical requirements. The institution must make electronic mail available to the target population, and 
the researcher must know how to send messages and work with in-coming mail. Developing the distribution list 
may require working with computer center professionals. 

The survey’s topic and predominant question format can also cause one method to have merits relative to 
the other. HTML forms are optimal for closed-ended items, while distribution list surveys are optimal for open- 
ended items. Extremely sensitive topics should probably stay off the internet entirely. An HTML form would be 
much easier to read from across the room than an electronic mail message, so slightly sensitive issues might be 
better suited to the distribution list method. On the other hand, when students respond to a question posted to a 
distribution list, their electronic mail response will contain their own address. It is much easier to guarantee 
complete anonymity through an HTML form submission, which automatically identifies the name of the file server 
as the sender of the message. 

Because the distribution list method returns data that automatically contains the student’s address, tracking 
respondents, doing second waves, and merging the survey results with other information in the institution’s data 
base is possible. HTML surveys are not done in waves. If the researcher must compare attributes of respondents to 
the population at large in order to evaluate potential biases, the HTML form must ask respondents to identify 
themselves. 

Sample bias represents one problem area that affects both of these methods. Clearly, those who read and 
respond to electronic mail do not represent the population of all students. And, it is probably safe to say that more 
people have access to electronic mail than to Web Pages: learning how to read your electronic mail is one skill; 
learning how to surf the Web is another thing entirely. An institution might end up with business, computer 
science and public policy majors over-represented in the results, and art, music and theater majors almost absent. 
Other potential biases might also warrant concern. For example, a 1996 mail survey of non-traditional aged 
students at the University of Wisconsin - Green Bay found that over half of them had never used electronic mail or 




accessed the internet. 
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The title of this paper, “Between Anecdote and Science”, suggests that electronic mail surveys can produce 
data for an organization that is much more complete, in terms of reliability, validity and representativeness, than the 
anecdotal evidence we must often use. Because the technology is new to researchers and new to participants, it 
seems unwise to expect electronic mail surveying to produce results that equal the reliability, validity and 
representativeness of traditional methods (see table 4). As internet access expands and people grow increasingly 
comfortable with electronic means of communications, many of the more obvious sample biases should dissipate. 



Table 4. Electronic Mai! Surveying vs. Traditional Methods 



Research Goals 


Electronic Mai! 
Surveys 


Traditional Survey 
Methods 


Save Money 


Yes 


No 


Save Time 


Yes 


No 


Maximize Representativeness 


No 


Yes 



In the past, researchers might have hesitated to develop a survey to evaluate a small, new or ad-hoc 
program because of the time and costs involved. Electronic mail surveying offers many potential time and material 
savings: there are no surveys to duplicate or fold, no envelopes to purchase, no mailings to stuff, no stamps to 
purchase and affix, no zip codes to sort, and no data to key. A researcher could design and pilot an instrument in 
one week, administer it the following week, do a follow-up in week three, and analyze the results in the fourth week. 
Organizations that have relied on anecdotal evidence as a primary way of evaluating certain programs because it was 
not feasible to do a long and costly mail survey or labor intensive phone or face interview can use electronic 
surveying to gather more generalizable conclusions about that program’s merits. The data might not be quite as 
representative as from a mail survey, but they would certainly be more representative than no data at all. 



Between Anecdote and Science 

19 



References 

Carr, H.H. (1991). Is using computer-based questionnaires better than using paper? Journal of Systems 
Management , 19, 37. 

Grobe, M. (1996). HTML Quick Reference. Academic Computing Services, The University of Kansas. 
[http://www.cc.ukans.edu/info/HTML_quick.html] 

Kiesler, S. and Sproull, L.S. (1986). Response effects in the electronic survey. Public Opinion Quarterly 50, 402- 
13. 

National Center for Supercomputing Applications. (1996). A beginner’s Guide to HTML . University of Illinois at 
Urbana-Champaign. [http://www.ncsa.uiuc.edu/General/Intemet/WWW/HTMLPrimerAll.html] 

Rosenfeld, Paul, et. al. (1993). Computer-administered surveys in organizational settings: alternatives, advantages, 
and applications. American Behavioral Scientist , 36(4), 485-5 1 1 . 

Sproull, L.S. (1986). Using electronic mail for data collection in organizational research. Academy of Management 
Journal , 29, 159-69. 

Stienfield, C.W. (1990). Computer-mediated communications in the organization: using electronic mail at Xerox. 

In C.W. Stienfield (Ed.), Organizational and Communication Theory . Newbury Park, CA: Sage. 

Synodinos, N.C., and Brennan, J.M. (1988). Computer interactive interviewing in survey research. Psychology 
and Marketing, 5(2), 117-37. 

Thach, L. (1995). Using e-mail to conduct survey research. Educational Technology , 35, 37-31. 

Walsh, J.P., Kiesler, S., Sproull, L.S., and Hesse, B.W. (1992). Self-selected and randomly selected respondents 
in a computer network survey. Public Opinion Quarterly , 56, 241-44. 



21 



o 

ERLC 





U S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 




NOTICE 

REPRODUCTION BASIS 




This document is covered by a signed “Reproduction Release 
(Blanket)” form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release 
form (either “Specific Document” or “Blanket”). 



0 




(9/92) 



